{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-Tuning a BERT Model and Create a Text Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let’s fine-tune the BERT model to our Customer Reviews Dataset and add a new classification layer to predict the `star_rating` for a given `review_body`.\n",
    "\n",
    "As mentioned earlier, BERT’s attention mechanism is called a Transformer. This is, not coincidentally, the name of the popular BERT Python library, “Transformers,” maintained by a company called HuggingFace. We will use a variant of BERT called [DistilBert](https://arxiv.org/pdf/1910.01108.pdf) which requires less memory and compute, but maintains very good accuracy on our dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous section, we've already performed the Feature Engineering to create BERT embeddings from the `reviews_body` text using the pre-trained BERT model, and split the dataset into train, validation and test files. To optimize for Tensorflow training, we saved the files in TFRecord format. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BERT Training](img/bert_training.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![BERT Pre-Processing](img/prepare_dataset_bert.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --user -qU 'sagemaker[local]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "ClientError",
     "evalue": "An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mClientError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-c7b864f8e431>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0msess\u001b[0m   \u001b[0;34m=\u001b[0m \u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mbucket\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault_bucket\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0;31m#role = sagemaker.get_execution_role()\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0mrole\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrole\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mdefault_bucket\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    368\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mdefault_bucket\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    369\u001b[0m             account = self.boto_session.client(\n\u001b[0;32m--> 370\u001b[0;31m                 \u001b[0;34m\"sts\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mregion_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendpoint_url\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msts_regional_endpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    371\u001b[0m             ).get_caller_identity()[\"Account\"]\n\u001b[1;32m    372\u001b[0m             \u001b[0mdefault_bucket\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"sagemaker-{}-{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccount\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    314\u001b[0m                     \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m    315\u001b[0m             \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 316\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    317\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    318\u001b[0m         \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m    633\u001b[0m             \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    634\u001b[0m             \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 635\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    636\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    637\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mClientError\u001b[0m: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid."
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "#role = sagemaker.get_execution_role()\n",
    "role = role = 'arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881'\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify the Dataset\n",
    "We are using the train, validation, and test splits created in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: ./data-tfrecord/: No such file or directory\r\n"
     ]
    }
   ],
   "source": [
    "!ls ./data-tfrecord/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file://data-tfrecord/bert-train\n"
     ]
    }
   ],
   "source": [
    "processed_train_data_local = 'file://data-tfrecord/bert-train'\n",
    "print(processed_train_data_local)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file://data-tfrecord/bert-train\n"
     ]
    }
   ],
   "source": [
    "processed_validation_data_local = 'file://data-tfrecord/bert-train'\n",
    "print(processed_validation_data_local)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file://data-tfrecord/bert-train\n"
     ]
    }
   ],
   "source": [
    "processed_test_data_local = 'file://data-tfrecord/bert-train'\n",
    "print(processed_validation_data_local)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show TensorFlow Training Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtime\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mrandom\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mpandas\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mpd\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mglob\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m glob\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mpprint\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36margparse\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msubprocess\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msys\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mtf\u001b[39;49;00m\n",
      "\u001b[37m#subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'tensorflow==2.1.0'])\u001b[39;49;00m\n",
      "subprocess.check_call([sys.executable, \u001b[33m'\u001b[39;49;00m\u001b[33m-m\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33mpip\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33minstall\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33mtransformers==2.8.0\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "subprocess.check_call([sys.executable, \u001b[33m'\u001b[39;49;00m\u001b[33m-m\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33mpip\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33minstall\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33msagemaker-tensorflow==2.1.0.1.0.0\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "subprocess.check_call([sys.executable, \u001b[33m'\u001b[39;49;00m\u001b[33m-m\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33mpip\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33minstall\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33msmdebug==0.7.2\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m DistilBertTokenizer\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m TFDistilBertForSequenceClassification\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m TextClassificationPipeline\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mconfiguration_distilbert\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m DistilBertConfig\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mkeras\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mcallbacks\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m ModelCheckpoint\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mkeras\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mmodels\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m load_model\n",
      "\n",
      "\n",
      "\n",
      "CLASSES = [\u001b[34m1\u001b[39;49;00m, \u001b[34m2\u001b[39;49;00m, \u001b[34m3\u001b[39;49;00m, \u001b[34m4\u001b[39;49;00m, \u001b[34m5\u001b[39;49;00m]\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mselect_data_and_label_from_record\u001b[39;49;00m(record):\n",
      "    x = {\n",
      "        \u001b[33m'\u001b[39;49;00m\u001b[33minput_ids\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m: record[\u001b[33m'\u001b[39;49;00m\u001b[33minput_ids\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m],\n",
      "        \u001b[33m'\u001b[39;49;00m\u001b[33minput_mask\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m: record[\u001b[33m'\u001b[39;49;00m\u001b[33minput_mask\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m],\n",
      "        \u001b[33m'\u001b[39;49;00m\u001b[33msegment_ids\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m: record[\u001b[33m'\u001b[39;49;00m\u001b[33msegment_ids\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]\n",
      "    }\n",
      "\n",
      "    y = record[\u001b[33m'\u001b[39;49;00m\u001b[33mlabel_ids\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m (x, y)\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mfile_based_input_dataset_builder\u001b[39;49;00m(channel,\n",
      "                                     input_filenames,\n",
      "                                     pipe_mode,\n",
      "                                     is_training,\n",
      "                                     drop_remainder,\n",
      "                                     batch_size,\n",
      "                                     epochs,\n",
      "                                     steps_per_epoch,\n",
      "                                     max_seq_length):\n",
      "\n",
      "    \u001b[37m# For training, we want a lot of parallel reading and shuffling.\u001b[39;49;00m\n",
      "    \u001b[37m# For eval, we want no shuffling and parallel reading doesn't matter.\u001b[39;49;00m\n",
      "\n",
      "    \u001b[34mif\u001b[39;49;00m pipe_mode:\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33m***** Using pipe_mode with channel \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(channel))\n",
      "        \u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msagemaker_tensorflow\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m PipeModeDataset\n",
      "        dataset = PipeModeDataset(channel=channel,\n",
      "                                  record_format=\u001b[33m'\u001b[39;49;00m\u001b[33mTFRecord\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    \u001b[34melse\u001b[39;49;00m:\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33m***** Using input_filenames \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(input_filenames))\n",
      "        dataset = tf.data.TFRecordDataset(input_filenames)\n",
      "\n",
      "    dataset = dataset.repeat(epochs * steps_per_epoch)\n",
      "    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
      "\n",
      "    name_to_features = {\n",
      "      \u001b[33m\"\u001b[39;49;00m\u001b[33minput_ids\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: tf.io.FixedLenFeature([max_seq_length], tf.int64),\n",
      "      \u001b[33m\"\u001b[39;49;00m\u001b[33minput_mask\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: tf.io.FixedLenFeature([max_seq_length], tf.int64),\n",
      "      \u001b[33m\"\u001b[39;49;00m\u001b[33msegment_ids\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: tf.io.FixedLenFeature([max_seq_length], tf.int64),\n",
      "      \u001b[33m\"\u001b[39;49;00m\u001b[33mlabel_ids\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: tf.io.FixedLenFeature([], tf.int64),\n",
      "    }\n",
      "\n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32m_decode_record\u001b[39;49;00m(record, name_to_features):\n",
      "        \u001b[33m\"\"\"Decodes a record to a TensorFlow example.\"\"\"\u001b[39;49;00m\n",
      "        record = tf.io.parse_single_example(record, name_to_features)\n",
      "        \u001b[37m# TODO:  wip/bert/bert_attention_head_view/train.py\u001b[39;49;00m\n",
      "        \u001b[37m# Convert input_ids into input_tokens with DistilBert vocabulary \u001b[39;49;00m\n",
      "        \u001b[37m#  if hook.get_collections()['all'].save_config.should_save_step(modes.EVAL, hook.mode_steps[modes.EVAL]):\u001b[39;49;00m\n",
      "        \u001b[37m#    hook._write_raw_tensor_simple(\"input_tokens\", input_tokens)\u001b[39;49;00m\n",
      "        \u001b[34mreturn\u001b[39;49;00m record\n",
      "    \n",
      "    dataset = dataset.apply(\n",
      "        tf.data.experimental.map_and_batch(\n",
      "          \u001b[34mlambda\u001b[39;49;00m record: _decode_record(record, name_to_features),\n",
      "          batch_size=batch_size,\n",
      "          drop_remainder=drop_remainder,\n",
      "          num_parallel_calls=tf.data.experimental.AUTOTUNE))\n",
      "\n",
      "    dataset.cache()\n",
      "\n",
      "    \u001b[34mif\u001b[39;49;00m is_training:\n",
      "        dataset = dataset.shuffle(seed=\u001b[34m42\u001b[39;49;00m,\n",
      "                                  buffer_size=\u001b[34m1000\u001b[39;49;00m,\n",
      "                                  reshuffle_each_iteration=\u001b[34mTrue\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m dataset\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mload_checkpoint_model\u001b[39;49;00m(checkpoint_path):\n",
      "    \u001b[37m# TODO:  find the latest checkpoint file\u001b[39;49;00m\n",
      "    max_epoch_filename = \u001b[34m1\u001b[39;49;00m \n",
      "    max_epoch_number = \u001b[34m1\u001b[39;49;00m \n",
      "\n",
      "    loaded_model = load_model(\u001b[33m'\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m/bert-checkpoint-001.h5\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(checkpoint_path))\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m loaded_model, max_epoch_number\n",
      "\n",
      "\n",
      "\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m'\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m:\n",
      "    parser = argparse.ArgumentParser()\n",
      "\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--train_data\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, \n",
      "                        default=os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_CHANNEL_TRAIN\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--validation_data\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, \n",
      "                        default=os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_CHANNEL_VALIDATION\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--test_data\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m,\n",
      "                        default=os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_CHANNEL_TEST\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "    \u001b[37m# This points to the S3 location - this should not be used by our code\u001b[39;49;00m\n",
      "    \u001b[37m# We should use /opt/ml/model/ instead\u001b[39;49;00m\n",
      "\u001b[37m#     parser.add_argument('--model_dir', \u001b[39;49;00m\n",
      "\u001b[37m#                         type=str, \u001b[39;49;00m\n",
      "\u001b[37m#                         default=os.environ['SM_MODEL_DIR'])\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--output_dir\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m,\n",
      "                        default=os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_OUTPUT_DIR\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "    \u001b[37m# This is unused\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--output_data_dir\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m,\n",
      "                        default=os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_OUTPUT_DATA_DIR\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--hosts\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mlist\u001b[39;49;00m, \n",
      "                        default=json.loads(os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_HOSTS\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]))\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--current_host\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, \n",
      "                        default=os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_CURRENT_HOST\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])    \n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--num_gpus\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, \n",
      "                        default=os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_NUM_GPUS\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--checkpoint_path\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, \n",
      "                        default=\u001b[33m'\u001b[39;49;00m\u001b[33m/opt/ml/checkpoints\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--use_xla\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36meval\u001b[39;49;00m,\n",
      "                        default=\u001b[34mFalse\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--use_amp\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36meval\u001b[39;49;00m,\n",
      "                        default=\u001b[34mFalse\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--max_seq_length\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m128\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--train_batch_size\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m128\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--validation_batch_size\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m256\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--test_batch_size\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m256\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--epochs\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m2\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--learning_rate\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mfloat\u001b[39;49;00m,\n",
      "                        default=\u001b[34m0.00003\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--epsilon\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mfloat\u001b[39;49;00m,\n",
      "                        default=\u001b[34m0.00000001\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--train_steps_per_epoch\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m1000\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--validation_steps\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m1000\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--test_steps\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "                        default=\u001b[34m1000\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--freeze_bert_layer\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36meval\u001b[39;49;00m,\n",
      "                        default=\u001b[34mFalse\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--enable_sagemaker_debugger\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36meval\u001b[39;49;00m,\n",
      "                        default=\u001b[34mFalse\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--run_validation\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36meval\u001b[39;49;00m,\n",
      "                        default=\u001b[34mFalse\u001b[39;49;00m)    \n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--run_test\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36meval\u001b[39;49;00m,\n",
      "                        default=\u001b[34mFalse\u001b[39;49;00m)    \n",
      "    parser.add_argument(\u001b[33m'\u001b[39;49;00m\u001b[33m--run_sample_predictions\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                        \u001b[36mtype\u001b[39;49;00m=\u001b[36meval\u001b[39;49;00m,\n",
      "                        default=\u001b[34mFalse\u001b[39;49;00m)\n",
      "    \n",
      "    args, _ = parser.parse_known_args()\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mArgs:\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m) \n",
      "    \u001b[36mprint\u001b[39;49;00m(args)\n",
      "    \n",
      "    env_var = os.environ \n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mEnvironment Variables:\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m) \n",
      "    pprint.pprint(\u001b[36mdict\u001b[39;49;00m(env_var), width = \u001b[34m1\u001b[39;49;00m) \n",
      "\n",
      "    train_data = args.train_data\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtrain_data \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(train_data))\n",
      "    validation_data = args.validation_data\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mvalidation_data \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(validation_data))\n",
      "    test_data = args.test_data\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtest_data \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(test_data))    \n",
      "\n",
      "\u001b[37m#    model_dir = args.model_dir\u001b[39;49;00m\n",
      "\u001b[37m#    print('model_dir {}'.format(model_dir))    \u001b[39;49;00m\n",
      "    local_model_dir = os.environ[\u001b[33m'\u001b[39;49;00m\u001b[33mSM_MODEL_DIR\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]\n",
      "\n",
      "    output_dir = args.output_dir\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33moutput_dir \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(output_dir))    \n",
      "\n",
      "    \u001b[37m# This is unused\u001b[39;49;00m\n",
      "\u001b[37m#    output_data_dir = args.output_data_dir\u001b[39;49;00m\n",
      "\u001b[37m#    print('output_data_dir {}'.format(output_data_dir))    \u001b[39;49;00m\n",
      "    hosts = args.hosts\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mhosts \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(hosts))    \n",
      "    current_host = args.current_host\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mcurrent_host \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(current_host))    \n",
      "    num_gpus = args.num_gpus\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mnum_gpus \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(num_gpus))\n",
      "    use_xla = args.use_xla\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33muse_xla \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(use_xla))    \n",
      "    use_amp = args.use_amp\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33muse_amp \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(use_amp))    \n",
      "    max_seq_length = args.max_seq_length\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mmax_seq_length \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(max_seq_length))    \n",
      "    train_batch_size = args.train_batch_size\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtrain_batch_size \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(train_batch_size))    \n",
      "    validation_batch_size = args.validation_batch_size\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mvalidation_batch_size \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(validation_batch_size))    \n",
      "    test_batch_size = args.test_batch_size\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtest_batch_size \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(test_batch_size))    \n",
      "    epochs = args.epochs\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mepochs \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(epochs))    \n",
      "    learning_rate = args.learning_rate\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mlearning_rate \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(learning_rate))    \n",
      "    epsilon = args.epsilon\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mepsilon \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(epsilon))    \n",
      "    train_steps_per_epoch = args.train_steps_per_epoch\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtrain_steps_per_epoch \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(train_steps_per_epoch))    \n",
      "    validation_steps = args.validation_steps\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mvalidation_steps \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(validation_steps))    \n",
      "    test_steps = args.test_steps\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtest_steps \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(test_steps))    \n",
      "    freeze_bert_layer = args.freeze_bert_layer\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mfreeze_bert_layer \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(freeze_bert_layer))    \n",
      "    enable_sagemaker_debugger = args.enable_sagemaker_debugger\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33menable_sagemaker_debugger \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(enable_sagemaker_debugger))    \n",
      "    run_validation = args.run_validation\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mrun_validation \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(run_validation))    \n",
      "    run_test = args.run_test\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mrun_test \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(run_test))    \n",
      "    run_sample_predictions = args.run_sample_predictions\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mrun_sample_predictions \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(run_sample_predictions))    \n",
      "\n",
      "    checkpoint_path = args.checkpoint_path\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mcheckpoint_path \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(checkpoint_path))\n",
      "    \n",
      "    \u001b[37m# Determine if PipeMode is enabled \u001b[39;49;00m\n",
      "    pipe_mode_str = os.environ.get(\u001b[33m'\u001b[39;49;00m\u001b[33mSM_INPUT_DATA_CONFIG\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m, \u001b[33m'\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    pipe_mode = (pipe_mode_str.find(\u001b[33m'\u001b[39;49;00m\u001b[33mPipe\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m) >= \u001b[34m0\u001b[39;49;00m)\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mUsing pipe_mode: \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(pipe_mode))\n",
      " \n",
      "    \u001b[37m# Model Output \u001b[39;49;00m\n",
      "    transformer_fine_tuned_model_path = os.path.join(local_model_dir, \u001b[33m'\u001b[39;49;00m\u001b[33mtransformers/fine-tuned/\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    os.makedirs(transformer_fine_tuned_model_path, exist_ok=\u001b[34mTrue\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[37m# SavedModel Output\u001b[39;49;00m\n",
      "    tensorflow_saved_model_path = os.path.join(local_model_dir, \u001b[33m'\u001b[39;49;00m\u001b[33mtensorflow/saved_model/0\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    os.makedirs(tensorflow_saved_model_path, exist_ok=\u001b[34mTrue\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[37m# Tensorboard Logs \u001b[39;49;00m\n",
      "    tensorboard_logs_path = os.path.join(output_dir, \u001b[33m'\u001b[39;49;00m\u001b[33mtensorboard\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m) \n",
      "    os.makedirs(tensorboard_logs_path, exist_ok=\u001b[34mTrue\u001b[39;49;00m)\n",
      "\n",
      "    distributed_strategy = tf.distribute.MirroredStrategy()\n",
      "    \u001b[37m# smdebug currently (0.7.2) does not support MultiWorkerMirroredStrategy()\u001b[39;49;00m\n",
      "    \u001b[37m# distributed_strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\u001b[39;49;00m\n",
      "    \u001b[34mwith\u001b[39;49;00m distributed_strategy.scope():\n",
      "        tf.config.optimizer.set_jit(use_xla)\n",
      "        tf.config.optimizer.set_experimental_options({\u001b[33m\"\u001b[39;49;00m\u001b[33mauto_mixed_precision\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: use_amp})\n",
      "\n",
      "        train_data_filenames = glob(os.path.join(train_data, \u001b[33m'\u001b[39;49;00m\u001b[33m*.tfrecord\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m))\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtrain_data_filenames \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(train_data_filenames))\n",
      "        train_dataset = file_based_input_dataset_builder(\n",
      "            channel=\u001b[33m'\u001b[39;49;00m\u001b[33mtrain\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "            input_filenames=train_data_filenames,\n",
      "            pipe_mode=pipe_mode,\n",
      "            is_training=\u001b[34mTrue\u001b[39;49;00m,\n",
      "            drop_remainder=\u001b[34mFalse\u001b[39;49;00m,\n",
      "            batch_size=train_batch_size,\n",
      "            epochs=epochs,\n",
      "            steps_per_epoch=train_steps_per_epoch,\n",
      "            max_seq_length=max_seq_length).map(select_data_and_label_from_record)\n",
      "\n",
      "        tokenizer = \u001b[34mNone\u001b[39;49;00m\n",
      "        config = \u001b[34mNone\u001b[39;49;00m\n",
      "        model = \u001b[34mNone\u001b[39;49;00m\n",
      "\n",
      "        \u001b[37m# This is required when launching many instances at once...  the urllib request seems to get denied periodically\u001b[39;49;00m\n",
      "        successful_download = \u001b[34mFalse\u001b[39;49;00m\n",
      "        retries = \u001b[34m0\u001b[39;49;00m\n",
      "        \u001b[34mwhile\u001b[39;49;00m (retries < \u001b[34m5\u001b[39;49;00m \u001b[35mand\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m successful_download):\n",
      "            \u001b[34mtry\u001b[39;49;00m:\n",
      "                tokenizer = DistilBertTokenizer.from_pretrained(\u001b[33m'\u001b[39;49;00m\u001b[33mdistilbert-base-uncased\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "                config = DistilBertConfig.from_pretrained(\u001b[33m'\u001b[39;49;00m\u001b[33mdistilbert-base-uncased\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                                                          num_labels=\u001b[36mlen\u001b[39;49;00m(CLASSES))\n",
      "                model = TFDistilBertForSequenceClassification.from_pretrained(\u001b[33m'\u001b[39;49;00m\u001b[33mdistilbert-base-uncased\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                                                                              config=config)\n",
      "                successful_download = \u001b[34mTrue\u001b[39;49;00m\n",
      "                \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mSucessfully downloaded after \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m retries.\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(retries))\n",
      "            \u001b[34mexcept\u001b[39;49;00m:\n",
      "                retries = retries + \u001b[34m1\u001b[39;49;00m\n",
      "                random_sleep = random.randint(\u001b[34m1\u001b[39;49;00m, \u001b[34m30\u001b[39;49;00m)\n",
      "                \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mRetry #\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m.  Sleeping for \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m seconds\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(retries, random_sleep))\n",
      "                time.sleep(random_sleep)\n",
      "\n",
      "        os.makedirs(checkpoint_path, exist_ok=\u001b[34mTrue\u001b[39;49;00m)\n",
      "        \n",
      "\u001b[37m#        if os.listdir(checkpoint_path):\u001b[39;49;00m\n",
      "\u001b[37m#            print('***** Found checkpoint *****')\u001b[39;49;00m\n",
      "\u001b[37m#            print(checkpoint_path)\u001b[39;49;00m\n",
      "\u001b[37m#            model, epoch_number = load_checkpoint_model(checkpoint_path)\u001b[39;49;00m\n",
      "\u001b[37m#            print('***** Using checkpoint model {} *****'.format(model))\u001b[39;49;00m\n",
      "\n",
      "        \u001b[34mif\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m tokenizer \u001b[35mor\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m model \u001b[35mor\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m config:\n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mNot properly initialized...\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "\n",
      "        optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate, epsilon=epsilon)\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33m** use_amp \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(use_amp))        \n",
      "        \u001b[34mif\u001b[39;49;00m use_amp:\n",
      "            \u001b[37m# loss scaling is currently required when using mixed precision\u001b[39;49;00m\n",
      "            optimizer = tf.keras.mixed_precision.experimental.LossScaleOptimizer(optimizer, \u001b[33m'\u001b[39;49;00m\u001b[33mdynamic\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "\n",
      "        callbacks = []\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33menable_sagemaker_debugger \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(enable_sagemaker_debugger))\n",
      "        \u001b[34mif\u001b[39;49;00m enable_sagemaker_debugger:\n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33m*** DEBUGGING ***\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "            \u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msmdebug\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mtensorflow\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36msmd\u001b[39;49;00m\n",
      "            \u001b[37m# This assumes that we specified debugger_hook_config\u001b[39;49;00m\n",
      "            callback = smd.KerasHook.create_from_json_file()\n",
      "            \u001b[36mprint\u001b[39;49;00m(callback)\n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33m*** CALLBACK \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m ***\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(callback))\n",
      "            callbacks.append(callback)\n",
      "            optimizer = callback.wrap_optimizer(optimizer)\n",
      "\n",
      "        tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
      "                                                    log_dir=tensorboard_logs_path)\n",
      "        callbacks.append(tensorboard_callback)\n",
      "        \n",
      "        checkpoint_callback = ModelCheckpoint(\n",
      "            filepath=checkpoint_path + \u001b[33m'\u001b[39;49;00m\u001b[33m/bert-checkpoint-\u001b[39;49;00m\u001b[33m{epoch:03d}\u001b[39;49;00m\u001b[33m.h5\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "            save_weights_only=\u001b[34mFalse\u001b[39;49;00m,\n",
      "            verbose=\u001b[34m1\u001b[39;49;00m,\n",
      "            monitor=\u001b[33m'\u001b[39;49;00m\u001b[33mval_acc\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "        callbacks.append(checkpoint_callback)\n",
      "\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33m*** OPTIMIZER \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m ***\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(optimizer))\n",
      "        \n",
      "        loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=\u001b[34mTrue\u001b[39;49;00m)\n",
      "        metric = tf.keras.metrics.SparseCategoricalAccuracy(\u001b[33m'\u001b[39;49;00m\u001b[33maccuracy\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "\n",
      "        model.compile(optimizer=optimizer, loss=loss, metrics=[metric])\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mTrained model \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(model))\n",
      "                    \n",
      "        \u001b[36mprint\u001b[39;49;00m(model.summary())\n",
      "\n",
      "        \u001b[34mif\u001b[39;49;00m run_validation:\n",
      "            validation_data_filenames = glob(os.path.join(validation_data, \u001b[33m'\u001b[39;49;00m\u001b[33m*.tfrecord\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m))\n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mvalidation_data_filenames \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(validation_data_filenames))\n",
      "            validation_dataset = file_based_input_dataset_builder(\n",
      "                channel=\u001b[33m'\u001b[39;49;00m\u001b[33mvalidation\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                input_filenames=validation_data_filenames,\n",
      "                pipe_mode=pipe_mode,\n",
      "                is_training=\u001b[34mFalse\u001b[39;49;00m,\n",
      "                drop_remainder=\u001b[34mFalse\u001b[39;49;00m,\n",
      "                batch_size=validation_batch_size,\n",
      "                epochs=epochs,\n",
      "                steps_per_epoch=validation_steps,\n",
      "                max_seq_length=max_seq_length).map(select_data_and_label_from_record)\n",
      "            \n",
      "            \u001b[37m# HACK:  trim the Validation dataset down to equal the number of validation steps to workaround PipeMode issue\u001b[39;49;00m\n",
      "            validation_dataset = validation_dataset.take(validation_steps)\n",
      "            \n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mStarting Training and Validation...\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "            train_and_validation_history = model.fit(train_dataset,\n",
      "                                                     shuffle=\u001b[34mTrue\u001b[39;49;00m,\n",
      "                                                     epochs=epochs,\n",
      "                                                     steps_per_epoch=train_steps_per_epoch,\n",
      "                                                     validation_data=validation_dataset,\n",
      "                                                     validation_steps=validation_steps,\n",
      "                                                     callbacks=callbacks)\n",
      "            \u001b[36mprint\u001b[39;49;00m(train_and_validation_history)\n",
      "        \u001b[34melse\u001b[39;49;00m: \u001b[37m# Not running validation\u001b[39;49;00m\n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mStarting Training (Without Validation)...\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "            train_history = model.fit(train_dataset,\n",
      "                                      shuffle=\u001b[34mTrue\u001b[39;49;00m,\n",
      "                                      epochs=epochs,\n",
      "                                      steps_per_epoch=train_steps_per_epoch,\n",
      "                                      callbacks=callbacks)\n",
      "            \u001b[36mprint\u001b[39;49;00m(train_history)\n",
      "\n",
      "        \u001b[34mif\u001b[39;49;00m run_test:\n",
      "            test_data_filenames = glob(os.path.join(test_data, \u001b[33m'\u001b[39;49;00m\u001b[33m*.tfrecord\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m))\n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtest_data_filenames \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(test_data_filenames))\n",
      "            test_dataset = file_based_input_dataset_builder(\n",
      "                channel=\u001b[33m'\u001b[39;49;00m\u001b[33mtest\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                input_filenames=test_data_filenames,\n",
      "                pipe_mode=pipe_mode,\n",
      "                is_training=\u001b[34mFalse\u001b[39;49;00m,\n",
      "                drop_remainder=\u001b[34mFalse\u001b[39;49;00m,\n",
      "                batch_size=test_batch_size,\n",
      "                epochs=epochs,\n",
      "                steps_per_epoch=test_steps,\n",
      "                max_seq_length=max_seq_length).map(select_data_and_label_from_record)\n",
      "\n",
      "            \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mStarting test...\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "            test_history = model.evaluate(test_dataset,\n",
      "                                          steps=test_steps,\n",
      "                                          callbacks=callbacks)\n",
      "            \u001b[36mprint\u001b[39;49;00m(test_history)\n",
      "\n",
      "            \n",
      "        \u001b[37m# Save the fine-tuned Transformers Model\u001b[39;49;00m\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtransformer_fine_tuned_model_path \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(transformer_fine_tuned_model_path))   \n",
      "\n",
      "        model.save_pretrained(transformer_fine_tuned_model_path)\n",
      "\n",
      "        \u001b[37m# Save the TensorFlow SavedModel\u001b[39;49;00m\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mtensorflow_saved_model_path \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(tensorflow_saved_model_path))   \n",
      "        model.save(tensorflow_saved_model_path, save_format=\u001b[33m'\u001b[39;49;00m\u001b[33mtf\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[34mif\u001b[39;49;00m run_sample_predictions:\n",
      "        loaded_model = TFDistilBertForSequenceClassification.from_pretrained(transformer_fine_tuned_model_path,\n",
      "                                                                       id2label={\n",
      "                                                                        \u001b[34m0\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m1\u001b[39;49;00m: \u001b[34m2\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m2\u001b[39;49;00m: \u001b[34m3\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m3\u001b[39;49;00m: \u001b[34m4\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m4\u001b[39;49;00m: \u001b[34m5\u001b[39;49;00m\n",
      "                                                                       },\n",
      "                                                                       label2id={\n",
      "                                                                        \u001b[34m1\u001b[39;49;00m: \u001b[34m0\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m2\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m3\u001b[39;49;00m: \u001b[34m2\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m4\u001b[39;49;00m: \u001b[34m3\u001b[39;49;00m,\n",
      "                                                                        \u001b[34m5\u001b[39;49;00m: \u001b[34m4\u001b[39;49;00m\n",
      "                                                                       })\n",
      "\n",
      "        tokenizer = DistilBertTokenizer.from_pretrained(\u001b[33m'\u001b[39;49;00m\u001b[33mdistilbert-base-uncased\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "\n",
      "        \u001b[34mif\u001b[39;49;00m num_gpus >= \u001b[34m1\u001b[39;49;00m:\n",
      "            inference_device = \u001b[34m0\u001b[39;49;00m \u001b[37m# GPU 0\u001b[39;49;00m\n",
      "        \u001b[34melse\u001b[39;49;00m:\n",
      "            inference_device = -\u001b[34m1\u001b[39;49;00m \u001b[37m# CPU\u001b[39;49;00m\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33minference_device \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(inference_device))\n",
      "\n",
      "        inference_pipeline = TextClassificationPipeline(model=loaded_model, \n",
      "                                                        tokenizer=tokenizer,\n",
      "                                                        framework=\u001b[33m'\u001b[39;49;00m\u001b[33mtf\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\n",
      "                                                        device=inference_device)  \n",
      "\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\"\"\u001b[39;49;00m\u001b[33mI loved it!  I will recommend this to everyone.\u001b[39;49;00m\u001b[33m\"\"\"\u001b[39;49;00m, inference_pipeline(\u001b[33m\"\"\"\u001b[39;49;00m\u001b[33mI loved it!  I will recommend this to everyone.\u001b[39;49;00m\u001b[33m\"\"\"\u001b[39;49;00m))\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\"\"\u001b[39;49;00m\u001b[33mIt\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33ms OK.\u001b[39;49;00m\u001b[33m\"\"\"\u001b[39;49;00m, inference_pipeline(\u001b[33m\"\"\"\u001b[39;49;00m\u001b[33mIt\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33ms OK.\u001b[39;49;00m\u001b[33m\"\"\"\u001b[39;49;00m))\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\"\"\u001b[39;49;00m\u001b[33mReally bad.  I hope they don\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33mt make this anymore.\u001b[39;49;00m\u001b[33m\"\"\"\u001b[39;49;00m, inference_pipeline(\u001b[33m\"\"\"\u001b[39;49;00m\u001b[33mReally bad.  I hope they don\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33mt make this anymore.\u001b[39;49;00m\u001b[33m\"\"\"\u001b[39;49;00m))\n"
     ]
    }
   ],
   "source": [
    "!pygmentize src/tf_bert_reviews.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Hyper-Parameters for Classification Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs=1\n",
    "learning_rate=0.00001\n",
    "epsilon=0.00000001\n",
    "train_batch_size=128\n",
    "validation_batch_size=128\n",
    "test_batch_size=128\n",
    "train_steps_per_epoch=50\n",
    "validation_steps=50\n",
    "test_steps=50\n",
    "train_instance_count=1\n",
    "train_instance_type='local'\n",
    "train_volume_size=1024\n",
    "use_xla=True\n",
    "use_amp=True\n",
    "freeze_bert_layer=False\n",
    "enable_sagemaker_debugger=True                    \n",
    "input_mode='File'\n",
    "run_validation=True\n",
    "run_test=True\n",
    "run_sample_predictions=True\n",
    "max_seq_length=128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Our BERT + TensorFlow Script to Run Locally\n",
    "Prepare our TensorFlow model to run on your local machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "estimator = TensorFlow(entry_point='tf_bert_reviews.py', \n",
    "                       source_dir='src', # put requirements.txt in this directory and it gets picked up\n",
    "                       role=role,\n",
    "                       train_instance_count=train_instance_count, # Make sure you have at least this number of input files or the ShardedByS3Key distibution strategy will fail the job due to no data available\n",
    "                       train_instance_type=train_instance_type,\n",
    "                       train_volume_size=train_volume_size,\n",
    "                       train_max_wait=7200, # Seconds to wait for spot instances to become available\n",
    "                       py_version='py3',\n",
    "                       framework_version='2.1.0',\n",
    "                       hyperparameters={'epochs': epochs,\n",
    "                                        'learning_rate': learning_rate,\n",
    "                                        'epsilon': epsilon,\n",
    "                                        'train_batch_size': train_batch_size,\n",
    "                                        'validation_batch_size': validation_batch_size,\n",
    "                                        'test_batch_size': test_batch_size,                                             \n",
    "                                        'train_steps_per_epoch': train_steps_per_epoch,\n",
    "                                        'validation_steps': validation_steps,\n",
    "                                        'test_steps': test_steps,\n",
    "                                        'use_xla': use_xla,\n",
    "                                        'use_amp': use_amp,                                             \n",
    "                                        'max_seq_length': max_seq_length,\n",
    "                                        'freeze_bert_layer': freeze_bert_layer,\n",
    "                                        'run_validation': run_validation,\n",
    "                                        'run_test': run_test,\n",
    "                                        'run_sample_predictions': run_sample_predictions},\n",
    "                       input_mode=input_mode,                    \n",
    "                       train_max_run=7200 # max 2 hours * 60 minutes seconds per hour * 60 seconds per minute\n",
    "                      )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: Due to an issue with resourceconfig.json not being found by the TensorFlow 2.0.0+ SageMaker Deep Learning Docker container, we can't run this.  TF 1.15 containers are ok, it seems.\n",
    "\n",
    "We're seeing this error:\n",
    "```\n",
    "RuntimeError: Failed to run: ['docker-compose', '-f', '/private/var/folders/50/1dlms49d3013ybsdl_k9nph0m05pfl/T/tmp0xw_kh44/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'create_image_uri' will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n",
      "'create_image_uri' will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating tmp0xw_kh44_algo-1-1eri7_1 ... \n",
      "\u001b[1BAttaching to tmp0xw_kh44_algo-1-1eri7_12mdone\u001b[0m\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:33:34,486 sagemaker-containers INFO     Imported framework sagemaker_tensorflow_container.training\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:33:34,500 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:33:35,770 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:33:35,801 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:33:35,823 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:33:35,834 sagemaker-containers INFO     Invoking user script\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Training Env:\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"additional_framework_parameters\": {},\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"channel_input_dirs\": {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"train\": \"/opt/ml/input/data/train\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"validation\": \"/opt/ml/input/data/validation\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"test\": \"/opt/ml/input/data/test\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     },\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"current_host\": \"algo-1-1eri7\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"framework_module\": \"sagemaker_tensorflow_container.training:main\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"hosts\": [\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"algo-1-1eri7\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     ],\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"hyperparameters\": {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"epochs\": 1,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"learning_rate\": 1e-05,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"epsilon\": 1e-08,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"train_batch_size\": 128,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"validation_batch_size\": 128,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"test_batch_size\": 128,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"train_steps_per_epoch\": 50,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"validation_steps\": 50,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"test_steps\": 50,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"use_xla\": true,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"use_amp\": true,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"max_seq_length\": 128,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"freeze_bert_layer\": false,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"run_validation\": true,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"run_test\": true,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"run_sample_predictions\": true,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"model_dir\": \"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     },\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"input_data_config\": {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"train\": {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m             \"TrainingInputMode\": \"File\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         },\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"validation\": {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m             \"TrainingInputMode\": \"File\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         },\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"test\": {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m             \"TrainingInputMode\": \"File\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         }\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     },\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"input_dir\": \"/opt/ml/input\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"is_master\": true,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"job_name\": \"tensorflow-training-2020-06-01-18-33-29-531\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"log_level\": 20,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"master_hostname\": \"algo-1-1eri7\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"model_dir\": \"/opt/ml/model\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"module_dir\": \"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/source/sourcedir.tar.gz\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"module_name\": \"tf_bert_reviews_local\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"network_interface_name\": \"eth0\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"num_cpus\": 8,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"num_gpus\": 0,\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"output_dir\": \"/opt/ml/output\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"resource_config\": {\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"current_host\": \"algo-1-1eri7\",\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         \"hosts\": [\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m             \"algo-1-1eri7\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m         ]\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     },\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m     \"user_entry_point\": \"tf_bert_reviews_local.py\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m }\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Environment variables:\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HOSTS=[\"algo-1-1eri7\"]\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_NETWORK_INTERFACE_NAME=eth0\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HPS={\"epochs\":1,\"epsilon\":1e-08,\"freeze_bert_layer\":false,\"learning_rate\":1e-05,\"max_seq_length\":128,\"model_dir\":\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\",\"run_sample_predictions\":true,\"run_test\":true,\"run_validation\":true,\"test_batch_size\":128,\"test_steps\":50,\"train_batch_size\":128,\"train_steps_per_epoch\":50,\"use_amp\":true,\"use_xla\":true,\"validation_batch_size\":128,\"validation_steps\":50}\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_USER_ENTRY_POINT=tf_bert_reviews_local.py\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_FRAMEWORK_PARAMS={}\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_RESOURCE_CONFIG={\"current_host\":\"algo-1-1eri7\",\"hosts\":[\"algo-1-1eri7\"]}\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_INPUT_DATA_CONFIG={\"test\":{\"TrainingInputMode\":\"File\"},\"train\":{\"TrainingInputMode\":\"File\"},\"validation\":{\"TrainingInputMode\":\"File\"}}\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_OUTPUT_DATA_DIR=/opt/ml/output/data\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_CHANNELS=[\"test\",\"train\",\"validation\"]\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_CURRENT_HOST=algo-1-1eri7\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_MODULE_NAME=tf_bert_reviews_local\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_LOG_LEVEL=20\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_INPUT_DIR=/opt/ml/input\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_INPUT_CONFIG_DIR=/opt/ml/input/config\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_OUTPUT_DIR=/opt/ml/output\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_NUM_CPUS=8\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_NUM_GPUS=0\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_MODEL_DIR=/opt/ml/model\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_MODULE_DIR=s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/source/sourcedir.tar.gz\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"test\":\"/opt/ml/input/data/test\",\"train\":\"/opt/ml/input/data/train\",\"validation\":\"/opt/ml/input/data/validation\"},\"current_host\":\"algo-1-1eri7\",\"framework_module\":\"sagemaker_tensorflow_container.training:main\",\"hosts\":[\"algo-1-1eri7\"],\"hyperparameters\":{\"epochs\":1,\"epsilon\":1e-08,\"freeze_bert_layer\":false,\"learning_rate\":1e-05,\"max_seq_length\":128,\"model_dir\":\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\",\"run_sample_predictions\":true,\"run_test\":true,\"run_validation\":true,\"test_batch_size\":128,\"test_steps\":50,\"train_batch_size\":128,\"train_steps_per_epoch\":50,\"use_amp\":true,\"use_xla\":true,\"validation_batch_size\":128,\"validation_steps\":50},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"test\":{\"TrainingInputMode\":\"File\"},\"train\":{\"TrainingInputMode\":\"File\"},\"validation\":{\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"tensorflow-training-2020-06-01-18-33-29-531\",\"log_level\":20,\"master_hostname\":\"algo-1-1eri7\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/source/sourcedir.tar.gz\",\"module_name\":\"tf_bert_reviews_local\",\"network_interface_name\":\"eth0\",\"num_cpus\":8,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1-1eri7\",\"hosts\":[\"algo-1-1eri7\"]},\"user_entry_point\":\"tf_bert_reviews_local.py\"}\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_USER_ARGS=[\"--epochs\",\"1\",\"--epsilon\",\"1e-08\",\"--freeze_bert_layer\",\"False\",\"--learning_rate\",\"1e-05\",\"--max_seq_length\",\"128\",\"--model_dir\",\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\",\"--run_sample_predictions\",\"True\",\"--run_test\",\"True\",\"--run_validation\",\"True\",\"--test_batch_size\",\"128\",\"--test_steps\",\"50\",\"--train_batch_size\",\"128\",\"--train_steps_per_epoch\",\"50\",\"--use_amp\",\"True\",\"--use_xla\",\"True\",\"--validation_batch_size\",\"128\",\"--validation_steps\",\"50\"]\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_CHANNEL_TRAIN=/opt/ml/input/data/train\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_CHANNEL_VALIDATION=/opt/ml/input/data/validation\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_CHANNEL_TEST=/opt/ml/input/data/test\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_EPOCHS=1\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_LEARNING_RATE=1e-05\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_EPSILON=1e-08\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_TRAIN_BATCH_SIZE=128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_VALIDATION_BATCH_SIZE=128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_TEST_BATCH_SIZE=128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_TRAIN_STEPS_PER_EPOCH=50\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_VALIDATION_STEPS=50\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_TEST_STEPS=50\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_USE_XLA=true\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_USE_AMP=true\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_MAX_SEQ_LENGTH=128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_FREEZE_BERT_LAYER=false\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_RUN_VALIDATION=true\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_RUN_TEST=true\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_RUN_SAMPLE_PREDICTIONS=true\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m SM_HP_MODEL_DIR=s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m PYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Invoking script with the following command:\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m /usr/bin/python3 tf_bert_reviews_local.py --epochs 1 --epsilon 1e-08 --freeze_bert_layer False --learning_rate 1e-05 --max_seq_length 128 --model_dir s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model --run_sample_predictions True --run_test True --run_validation True --test_batch_size 128 --test_steps 50 --train_batch_size 128 --train_steps_per_epoch 50 --use_amp True --use_xla True --validation_batch_size 128 --validation_steps 50\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Collecting transformers==2.8.0\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading transformers-2.8.0-py3-none-any.whl (563 kB)\n",
      "\u001b[K     |████████████████████████████████| 563 kB 673 kB/s eta 0:00:01\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[?25hRequirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (1.12.43)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Collecting tokenizers==0.5.2\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading tokenizers-0.5.2-cp36-cp36m-manylinux1_x86_64.whl (3.7 MB)\n",
      "\u001b[K     |████████████████████████████████| 3.7 MB 3.0 MB/s eta 0:00:01\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[?25hCollecting regex!=2019.12.17\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading regex-2020.5.14-cp36-cp36m-manylinux2010_x86_64.whl (675 kB)\n",
      "\u001b[K     |████████████████████████████████| 675 kB 4.0 MB/s eta 0:00:01\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[?25hCollecting sentencepiece\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.1 MB 4.7 MB/s eta 0:00:01\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[?25hCollecting dataclasses; python_version < \"3.7\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading dataclasses-0.7-py3-none-any.whl (18 kB)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Collecting tqdm>=4.27\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading tqdm-4.46.0-py2.py3-none-any.whl (63 kB)\n",
      "\u001b[K     |████████████████████████████████| 63 kB 3.4 MB/s eta 0:00:011\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (1.18.1)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Collecting filelock\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (2.22.0)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Collecting sacremoses\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Downloading sacremoses-0.0.43.tar.gz (883 kB)\n",
      "\u001b[K     |████████████████████████████████| 883 kB 5.3 MB/s eta 0:00:01\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[?25hRequirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.8.0) (0.9.5)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: botocore<1.16.0,>=1.15.43 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.8.0) (1.15.43)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.8.0) (0.3.3)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (3.0.4)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (1.25.9)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2.8)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2020.4.5.1)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (1.14.0)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (7.1.1)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (0.14.1)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.16.0,>=1.15.43->boto3->transformers==2.8.0) (2.8.1)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.16.0,>=1.15.43->boto3->transformers==2.8.0) (0.15.2)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Building wheels for collected packages: sacremoses\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Building wheel for sacremoses (setup.py) ... \u001b[?25ldone\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[?25h  Created wheel for sacremoses: filename=sacremoses-0.0.43-py3-none-any.whl size=893259 sha256=b2dedd164cd44ca220dda89340a441870bbe02f8201ce9da6836aeb9a2adbb05\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m   Stored in directory: /root/.cache/pip/wheels/49/25/98/cdea9c79b2d9a22ccc59540b1784b67f06b633378e97f58da2\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Successfully built sacremoses\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Installing collected packages: tokenizers, regex, sentencepiece, dataclasses, tqdm, filelock, sacremoses, transformers\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Successfully installed dataclasses-0.7 filelock-3.0.12 regex-2020.5.14 sacremoses-0.0.43 sentencepiece-0.1.91 tokenizers-0.5.2 tqdm-4.46.0 transformers-2.8.0\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[33mWARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: sagemaker-tensorflow==2.1.0.1.0.0 in /usr/local/lib/python3.6/dist-packages (2.1.0.1.0.0)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[33mWARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: smdebug==0.7.2 in /usr/local/lib/python3.6/dist-packages (0.7.2)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: numpy<2.0.0,>1.16.0 in /usr/local/lib/python3.6/dist-packages (from smdebug==0.7.2) (1.18.1)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from smdebug==0.7.2) (3.11.3)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: boto3>=1.10.32 in /usr/local/lib/python3.6/dist-packages (from smdebug==0.7.2) (1.12.43)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from smdebug==0.7.2) (20.3)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.0->smdebug==0.7.2) (1.14.0)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.0->smdebug==0.7.2) (46.1.3)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: botocore<1.16.0,>=1.15.43 in /usr/local/lib/python3.6/dist-packages (from boto3>=1.10.32->smdebug==0.7.2) (1.15.43)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3>=1.10.32->smdebug==0.7.2) (0.9.5)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3>=1.10.32->smdebug==0.7.2) (0.3.3)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->smdebug==0.7.2) (2.4.7)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: urllib3<1.26,>=1.20; python_version != \"3.4\" in /usr/local/lib/python3.6/dist-packages (from botocore<1.16.0,>=1.15.43->boto3>=1.10.32->smdebug==0.7.2) (1.25.9)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.16.0,>=1.15.43->boto3>=1.10.32->smdebug==0.7.2) (0.15.2)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.16.0,>=1.15.43->boto3>=1.10.32->smdebug==0.7.2) (2.8.1)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \u001b[33mWARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m /opt/ml/config/resourceconfig.json not found.  current_host is unknown.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Args:\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Namespace(checkpoint_path='/opt/ml/checkpoints', current_host='algo-1-1eri7', enable_sagemaker_debugger=False, epochs=1, epsilon=1e-08, freeze_bert_layer=False, hosts=['algo-1-1eri7'], learning_rate=1e-05, max_seq_length=128, num_gpus=0, output_data_dir='/opt/ml/output/data', output_dir='/opt/ml/output', run_sample_predictions=True, run_test=True, run_validation=True, test_batch_size=128, test_data='/opt/ml/input/data/test', test_steps=50, train_batch_size=128, train_data='/opt/ml/input/data/train', train_steps_per_epoch=50, use_amp=True, use_xla=True, validation_batch_size=128, validation_data='/opt/ml/input/data/validation', validation_steps=50)\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Environment Variables:\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m {'AWS_ACCESS_KEY_ID': 'AKIA54RGMMRRZJAQKO6G',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'AWS_REGION': 'us-east-1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'AWS_SECRET_ACCESS_KEY': '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',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'CURRENT_HOST': 'algo-1-1eri7',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'DEBCONF_NONINTERACTIVE_SEEN': 'true',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'DEBIAN_FRONTEND': 'noninteractive',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'HOME': '/root',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'HOSTNAME': '997847954e91',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'KMP_AFFINITY': 'granularity=fine,compact,1,0',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'KMP_BLOCKTIME': '1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'KMP_DUPLICATE_LIB_OK': 'True',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'KMP_INIT_AT_FORK': 'FALSE',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'KMP_SETTINGS': '0',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'LANG': 'C.UTF-8',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'LC_ALL': 'C.UTF-8',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'LD_LIBRARY_PATH': '/usr/local/openmpi/lib:',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'PATH': '/usr/local/openmpi/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'PYTHONDONTWRITEBYTECODE': '1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'PYTHONIOENCODING': 'UTF-8',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'PYTHONPATH': '/opt/ml/code:/usr/local/bin:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'PYTHONUNBUFFERED': '1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'S3_REGION': 'us-east-1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'S3_USE_HTTPS': '1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SAGEMAKER_JOB_NAME': 'tensorflow-training-2020-06-01-18-33-29-531',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SAGEMAKER_REGION': 'us-east-1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SAGEMAKER_TRAINING_MODULE': 'sagemaker_tensorflow_container.training:main',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_CHANNELS': '[\"test\",\"train\",\"validation\"]',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_CHANNEL_TEST': '/opt/ml/input/data/test',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_CHANNEL_TRAIN': '/opt/ml/input/data/train',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_CHANNEL_VALIDATION': '/opt/ml/input/data/validation',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_CURRENT_HOST': 'algo-1-1eri7',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_FRAMEWORK_MODULE': 'sagemaker_tensorflow_container.training:main',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_FRAMEWORK_PARAMS': '{}',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HOSTS': '[\"algo-1-1eri7\"]',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HPS': '{\"epochs\":1,\"epsilon\":1e-08,\"freeze_bert_layer\":false,\"learning_rate\":1e-05,\"max_seq_length\":128,\"model_dir\":\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\",\"run_sample_predictions\":true,\"run_test\":true,\"run_validation\":true,\"test_batch_size\":128,\"test_steps\":50,\"train_batch_size\":128,\"train_steps_per_epoch\":50,\"use_amp\":true,\"use_xla\":true,\"validation_batch_size\":128,\"validation_steps\":50}',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_EPOCHS': '1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_EPSILON': '1e-08',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_FREEZE_BERT_LAYER': 'false',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_LEARNING_RATE': '1e-05',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_MAX_SEQ_LENGTH': '128',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_MODEL_DIR': 's3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_RUN_SAMPLE_PREDICTIONS': 'true',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_RUN_TEST': 'true',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_RUN_VALIDATION': 'true',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_TEST_BATCH_SIZE': '128',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_TEST_STEPS': '50',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_TRAIN_BATCH_SIZE': '128',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_TRAIN_STEPS_PER_EPOCH': '50',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_USE_AMP': 'true',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_USE_XLA': 'true',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_VALIDATION_BATCH_SIZE': '128',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_HP_VALIDATION_STEPS': '50',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_INPUT_CONFIG_DIR': '/opt/ml/input/config',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_INPUT_DATA_CONFIG': '{\"test\":{\"TrainingInputMode\":\"File\"},\"train\":{\"TrainingInputMode\":\"File\"},\"validation\":{\"TrainingInputMode\":\"File\"}}',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_INPUT_DIR': '/opt/ml/input',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_LOG_LEVEL': '20',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_MODEL_DIR': '/opt/ml/model',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_MODULE_DIR': 's3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/source/sourcedir.tar.gz',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_MODULE_NAME': 'tf_bert_reviews_local',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_NETWORK_INTERFACE_NAME': 'eth0',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_NUM_CPUS': '8',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_NUM_GPUS': '0',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_OUTPUT_DATA_DIR': '/opt/ml/output/data',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_OUTPUT_DIR': '/opt/ml/output',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_OUTPUT_INTERMEDIATE_DIR': '/opt/ml/output/intermediate',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_RESOURCE_CONFIG': '{\"current_host\":\"algo-1-1eri7\",\"hosts\":[\"algo-1-1eri7\"]}',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_TRAINING_ENV': '{\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"test\":\"/opt/ml/input/data/test\",\"train\":\"/opt/ml/input/data/train\",\"validation\":\"/opt/ml/input/data/validation\"},\"current_host\":\"algo-1-1eri7\",\"framework_module\":\"sagemaker_tensorflow_container.training:main\",\"hosts\":[\"algo-1-1eri7\"],\"hyperparameters\":{\"epochs\":1,\"epsilon\":1e-08,\"freeze_bert_layer\":false,\"learning_rate\":1e-05,\"max_seq_length\":128,\"model_dir\":\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\",\"run_sample_predictions\":true,\"run_test\":true,\"run_validation\":true,\"test_batch_size\":128,\"test_steps\":50,\"train_batch_size\":128,\"train_steps_per_epoch\":50,\"use_amp\":true,\"use_xla\":true,\"validation_batch_size\":128,\"validation_steps\":50},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"test\":{\"TrainingInputMode\":\"File\"},\"train\":{\"TrainingInputMode\":\"File\"},\"validation\":{\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"tensorflow-training-2020-06-01-18-33-29-531\",\"log_level\":20,\"master_hostname\":\"algo-1-1eri7\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/source/sourcedir.tar.gz\",\"module_name\":\"tf_bert_reviews_local\",\"network_interface_name\":\"eth0\",\"num_cpus\":8,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1-1eri7\",\"hosts\":[\"algo-1-1eri7\"]},\"user_entry_point\":\"tf_bert_reviews_local.py\"}',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_USER_ARGS': '[\"--epochs\",\"1\",\"--epsilon\",\"1e-08\",\"--freeze_bert_layer\",\"False\",\"--learning_rate\",\"1e-05\",\"--max_seq_length\",\"128\",\"--model_dir\",\"s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model\",\"--run_sample_predictions\",\"True\",\"--run_test\",\"True\",\"--run_validation\",\"True\",\"--test_batch_size\",\"128\",\"--test_steps\",\"50\",\"--train_batch_size\",\"128\",\"--train_steps_per_epoch\",\"50\",\"--use_amp\",\"True\",\"--use_xla\",\"True\",\"--validation_batch_size\",\"128\",\"--validation_steps\",\"50\"]',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'SM_USER_ENTRY_POINT': 'tf_bert_reviews_local.py',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'TERM': 'xterm',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'TF_CPP_MIN_LOG_LEVEL': '1',\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m  'TRAINING_JOB_NAME': 'tensorflow-training-2020-06-01-18-33-29-531'}\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m train_data /opt/ml/input/data/train\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m validation_data /opt/ml/input/data/validation\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m test_data /opt/ml/input/data/test\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m output_dir /opt/ml/output\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m hosts ['algo-1-1eri7']\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m current_host algo-1-1eri7\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m num_gpus 0\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m use_xla True\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m use_amp True\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m max_seq_length 128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m train_batch_size 128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m validation_batch_size 128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m test_batch_size 128\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m epochs 1\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m learning_rate 1e-05\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m epsilon 1e-08\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m train_steps_per_epoch 50\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m validation_steps 50\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m test_steps 50\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m freeze_bert_layer False\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m enable_sagemaker_debugger False\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m run_validation True\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m run_test True\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m run_sample_predictions True\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m checkpoint_path /opt/ml/checkpoints\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Using pipe_mode: False\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m train_data_filenames ['/opt/ml/input/data/train/part-algo-4-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord', '/opt/ml/input/data/train/part-algo-2-amazon_reviews_us_Digital_Software_v1_00.tfrecord']\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m ***** Using input_filenames ['/opt/ml/input/data/train/part-algo-4-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord', '/opt/ml/input/data/train/part-algo-2-amazon_reviews_us_Digital_Software_v1_00.tfrecord']\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m WARNING:tensorflow:From tf_bert_reviews_local.py:95: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Instructions for updating:\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading: 100% 232k/232k [00:00<00:00, 460kB/s]  \n",
      "Downloading: 100% 442/442 [00:00<00:00, 412kB/s]\n",
      "Downloading: 100% 363M/363M [01:30<00:00, 4.00MB/s]    \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:35:21.696041: W tensorflow/python/util/util.cc:319] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Sucessfully downloaded after 0 retries.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m ** use_amp True\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m enable_sagemaker_debugger False\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m *** OPTIMIZER <tensorflow.python.keras.mixed_precision.experimental.loss_scale_optimizer.LossScaleOptimizer object at 0x7f08cc10df98> ***\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Trained model <transformers.modeling_tf_distilbert.TFDistilBertForSequenceClassification object at 0x7f08ec116320>\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Model: \"tf_distil_bert_for_sequence_classification\"\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m _________________________________________________________________\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Layer (type)                 Output Shape              Param #   \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m =================================================================\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m distilbert (TFDistilBertMain multiple                  66362880  \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m _________________________________________________________________\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m pre_classifier (Dense)       multiple                  590592    \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m _________________________________________________________________\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m classifier (Dense)           multiple                  3845      \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m _________________________________________________________________\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m dropout_19 (Dropout)         multiple                  0         \n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m =================================================================\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Total params: 66,957,317\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Trainable params: 66,957,317\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Non-trainable params: 0\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m _________________________________________________________________\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m None\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m validation_data_filenames ['/opt/ml/input/data/validation/part-algo-4-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord', '/opt/ml/input/data/validation/part-algo-2-amazon_reviews_us_Digital_Software_v1_00.tfrecord']\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m ***** Using input_filenames ['/opt/ml/input/data/validation/part-algo-4-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord', '/opt/ml/input/data/validation/part-algo-2-amazon_reviews_us_Digital_Software_v1_00.tfrecord']\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Starting Training and Validation...\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Train for 50 steps, validate for 50 steps\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:35:54.633864: W tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1892] No (suitable) GPUs detected, skipping auto_mixed_precision graph optimizer\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:35:55.909462: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1574] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:36:39.890387: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (mklcpu) ran out of memory trying to allocate 48.00MiB (rounded to 50331648).  Current allocation summary follows.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:36:39.895041: W tensorflow/core/common_runtime/bfc_allocator.cc:429] ***************_********************************************************************xxx*************\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:36:39.897417: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at mkl_batch_matmul_op.cc:126 : Resource exhausted: OOM when allocating tensor with shape[128,12,128,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:36:39.899706: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Resource exhausted: OOM when allocating tensor with shape[128,12,128,64] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator mklcpu\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \t [[{{node tf_distil_bert_for_sequence_classification/distilbert/transformer/layer_._0/attention/MatMul_1}}]]\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m \n",
      " 1/50 [..............................] - ETA: 44:18\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Epoch 00001: saving model to /opt/ml/checkpoints/bert-checkpoint-001.h5\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m 2020-06-01 18:36:43,870 sagemaker-containers ERROR    ExecuteUserScriptError:\n",
      "\u001b[36malgo-1-1eri7_1  |\u001b[0m Command \"/usr/bin/python3 tf_bert_reviews_local.py --epochs 1 --epsilon 1e-08 --freeze_bert_layer False --learning_rate 1e-05 --max_seq_length 128 --model_dir s3://sagemaker-us-east-1-954636985443/tensorflow-training-2020-06-01-18-33-29-531/model --run_sample_predictions True --run_test True --run_validation True --test_batch_size 128 --test_steps 50 --train_batch_size 128 --train_steps_per_epoch 50 --use_amp True --use_xla True --validation_batch_size 128 --validation_steps 50\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'upload_data' method will be deprecated in favor of 'S3Uploader' class (https://sagemaker.readthedocs.io/en/stable/s3.html#sagemaker.s3.S3Uploader) in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36mtmp0xw_kh44_algo-1-1eri7_1 exited with code 1\n",
      "\u001b[0mAborting on container exit...\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Failed to run: ['docker-compose', '-f', '/private/var/folders/50/1dlms49d3013ybsdl_k9nph0m05pfl/T/tmp0xw_kh44/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/local/image.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, input_data_config, output_data_config, hyperparameters, job_name)\u001b[0m\n\u001b[1;32m    160\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m             \u001b[0m_stream_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprocess\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mRuntimeError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/local/image.py\u001b[0m in \u001b[0;36m_stream_output\u001b[0;34m(process)\u001b[0m\n\u001b[1;32m    676\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mexit_code\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 677\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Process exited with code: %s\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mexit_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    678\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Process exited with code: 1",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-21-5224fb90ed2b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m                       \u001b[0;34m'test'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mprocessed_test_data_local\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m               },                             \n\u001b[0;32m----> 5\u001b[0;31m               wait=False)\n\u001b[0m",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/tensorflow/estimator.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, inputs, wait, logs, job_name, experiment_config, run_tensorboard_locally)\u001b[0m\n\u001b[1;32m    483\u001b[0m                 \u001b[0mtensorboard\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    484\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 485\u001b[0;31m             \u001b[0mfit_super\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    486\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    487\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mclassmethod\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/tensorflow/estimator.py\u001b[0m in \u001b[0;36mfit_super\u001b[0;34m()\u001b[0m\n\u001b[1;32m    462\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    463\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mfit_super\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 464\u001b[0;31m             \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTensorFlow\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlogs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexperiment_config\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    465\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    466\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrun_tensorboard_locally\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mwait\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, inputs, wait, logs, job_name, experiment_config)\u001b[0m\n\u001b[1;32m    492\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_prepare_for_training\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    493\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 494\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlatest_training_job\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_TrainingJob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_new\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexperiment_config\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    495\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjobs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlatest_training_job\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    496\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mstart_new\u001b[0;34m(cls, estimator, inputs, experiment_config)\u001b[0m\n\u001b[1;32m   1059\u001b[0m             \u001b[0mtrain_args\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"enable_sagemaker_metrics\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menable_sagemaker_metrics\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1060\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1061\u001b[0;31m         \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mtrain_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1062\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1063\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_current_job_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, input_mode, input_config, role, job_name, output_config, resource_config, vpc_config, hyperparameters, stop_condition, tags, metric_definitions, enable_network_isolation, image, algorithm_arn, encrypt_inter_container_traffic, train_use_spot_instances, checkpoint_s3_uri, checkpoint_local_path, experiment_config, debugger_rule_configs, debugger_hook_config, tensorboard_output_config, enable_sagemaker_metrics)\u001b[0m\n\u001b[1;32m    600\u001b[0m         \u001b[0mLOGGER\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Creating training-job with name: %s\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    601\u001b[0m         \u001b[0mLOGGER\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"train request: %s\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdumps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_request\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindent\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m4\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 602\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_training_job\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mtrain_request\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    603\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    604\u001b[0m     def process(\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/local/local_session.py\u001b[0m in \u001b[0;36mcreate_training_job\u001b[0;34m(self, TrainingJobName, AlgorithmSpecification, OutputDataConfig, ResourceConfig, InputDataConfig, **kwargs)\u001b[0m\n\u001b[1;32m    100\u001b[0m         \u001b[0mhyperparameters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"HyperParameters\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;34m\"HyperParameters\"\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m         \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Starting training job\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 102\u001b[0;31m         \u001b[0mtraining_job\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mInputDataConfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mOutputDataConfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhyperparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTrainingJobName\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    103\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m         \u001b[0mLocalSagemakerClient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_training_jobs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mTrainingJobName\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtraining_job\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/local/entities.py\u001b[0m in \u001b[0;36mstart\u001b[0;34m(self, input_data_config, output_data_config, hyperparameters, job_name)\u001b[0m\n\u001b[1;32m     94\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     95\u001b[0m         self.model_artifacts = self.container.train(\n\u001b[0;32m---> 96\u001b[0;31m             \u001b[0minput_data_config\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput_data_config\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhyperparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_name\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     97\u001b[0m         )\n\u001b[1;32m     98\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend_time\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/local/image.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, input_data_config, output_data_config, hyperparameters, job_name)\u001b[0m\n\u001b[1;32m    164\u001b[0m             \u001b[0;31m# which contains the exit code and append the command line to it.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m             \u001b[0mmsg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Failed to run: %s, %s\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mcompose_command\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m             \u001b[0martifacts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mretrieve_artifacts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcompose_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput_data_config\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Failed to run: ['docker-compose', '-f', '/private/var/folders/50/1dlms49d3013ybsdl_k9nph0m05pfl/T/tmp0xw_kh44/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1"
     ]
    }
   ],
   "source": [
    "estimator.fit(inputs={'train': processed_train_data_local, \n",
    "                      'validation': processed_validation_data_local,\n",
    "                      'test': processed_test_data_local\n",
    "              },                             \n",
    "              wait=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_job_name = estimator.latest_training_job.name\n",
    "print('Training Job Name:  {}'.format(training_job_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/jobs/{}\">Training Job</a> After About 5 Minutes</b>'.format(region, training_job_name)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a href=\"https://console.aws.amazon.com/cloudwatch/home?region={}#logStream:group=/aws/sagemaker/TrainingJobs;prefix={};streamFilter=typeLogStreamPrefix\">CloudWatch Logs</a> After About 5 Minutes</b>'.format(region, training_job_name)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a href=\"https://s3.console.aws.amazon.com/s3/buckets/{}/{}/?region={}&tab=overview\">S3 Output Data</a> After The Training Job Has Completed</b>'.format(bucket, training_job_name, region)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wait Until the ^^ Training Job ^^ Completes Above!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator.latest_training_job.wait(logs=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inspect the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (localsm)",
   "language": "python",
   "name": "localsm"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
